跳转至

NVIDIA Cosmos-H-Dreams：将实时生成式模拟引入手术机器人

文章背景与核心概要

手术机器人正从基础的远程操作向先进的视觉-语言-动作策略演进。然而，真实的机器人物理训练速度慢、成本高，且存在损坏精密设备或生物材料的风险。传统的模拟器很难准确建模复杂的手术环境，例如可变形组织、流体以及精细的器械交互。

为了克服这些挑战，NVIDIA 推出了 Cosmos-H-Dreams——一个专为手术机器人设计的实时、动作条件生成式模拟器。该系统通过将 Cosmos-H-Surgical-Simulator 蒸馏为因果学生模型，并与 NVIDIA 的 FlashDreams 流式推理库相结合，在单块 NVIDIA RTX PRO 6000 GPU 上实现了约 160 FPS 的交互式执行。这一突破赋予了开发者、研究人员以及经训练的策略在闭环环境中安全测试、评估和生成合成手术数据的能力。

📌 Summary

Surgical robotics is shifting from basic teleoperation toward advanced vision-language-action policies. However, physical robotic training is slow, expensive, and carries the risk of damaging delicate equipment or biological material. Traditional simulators struggle to accurately model complex surgical environments like deformable tissues, fluids, and fine instrument interactions.

To overcome this, NVIDIA introduces Cosmos-H-Dreams—a real-time, action-conditioned generative simulator for surgical robotics. Built by distilling the Cosmos-H-Surgical-Simulator into a causal student model and pairing it with NVIDIA’s FlashDreams streaming inference library, it achieves interactive execution (~160 FPS) on a single NVIDIA RTX PRO 6000 GPU. This breakthrough empowers developers, researchers, and trained policies to safely test, evaluate, and generate synthetic surgical data in a closed-loop environment.

1. From Surgical World Model to Interactive Simulator

Cosmos-H-Surgical-Simulator is an action-conditioned world foundation model built on NVIDIA Cosmos-Predict2.5-2B and trained on the Open-H-Embodiment dataset. Given an initial frame and a sequence of future robot actions, it predicts the visual consequences of those actions.

Cosmos-H-Dreams takes this concept into the real-time regime. Specializing the multi-embodiment surgical priors for da Vinci Research Kit (dVRK) tabletop suturing, it uses a causal student model that generates scenes autoregressively. Furthermore, collaborations with CMR Surgical and Cambridge Consultants have successfully integrated it with the Versius surgeon controller for real-time operation.

2. Distilling Cosmos-H-Surgical-Simulator for Real-Time Performance

To achieve low-latency generative simulation without sacrificing physical accuracy, Cosmos-H-Dreams relies on a rigorous teacher-student training pipeline:

2.1. A Surgical Teacher

  • Unified Action Space: Maps dual-arm dVRK actions (end-effector translation, rotation, and gripper state) into a 44-dimensional representation.
  • Failure Inclusion: Fine-tuned on successful demonstrations as well as failure cases (e.g., dropped needles, missed throws) to ensure policies can be evaluated on realistic negative outcomes.
  • Progressive Temporal Horizons: Training scales progressively from 12 frames up to 72 frames to ensure long-horizon stability.

2.2. Causal Warmup

A causal student model is initialized from the teacher and trained on precomputed, cached denoising trajectories to learn causal attention and streaming key/value caching.

2.3. Self-Forcing Distillation

To prevent error compounding during autoregressive rollouts, the student rolls forward using its own generated context, guided by distribution-matching supervision from a frozen teacher. This results in an efficient, few-step diffusion process requiring as few as two denoising steps per latent frame.

3. FlashDreams: The Real-Time Inference Engine

Cosmos-H-Dreams is powered by FlashDreams, an accelerated inference library that incorporates: * Streaming KV caches * CUDA Graph capturing * Advanced model compilation

These optimizations scale performance from ~10 frames per second up to ~160 frames per second on a single NVIDIA RTX PRO 6000 GPU.

Interactive Interfaces

  • Web Browser: Keyboard controls combined with WebRTC streaming.
  • Meta Quest (WebXR): Controller tracking mapped directly to robot actions in a synthesized immersive scene.
  • Policy Loops: Direct integration with learned surgical policies to cycle observations and predicted actions inside a closed loop.

4. Adapting to Your Own Data

Cosmos-H-Dreams provides a pre-trained checkpoint for tabletop suturing, but it is fully adaptable to custom robotic embodiments. NVIDIA provides a step-by-step teacher training and self-forcing guide to help researchers apply the framework to proprietary datasets.

5. What’s Next: Toward Closed-Loop Surgical Physical AI

Real-time generative simulation establishes new closed-loop evaluation paradigms: * Metrics: Tool-tip reach/pose accuracy, gripper-cycle fidelity, idle stability, and long-horizon drift tracking. * Training Efficiency: On-demand generation of rare failure states for reinforcement and imitation learning. * Future Horizons: Paving the way for latency-aware telesurgery support, interactive surgical rehearsal, and advanced intraoperative decision support systems.

6. Get Started Today